Make DXC and DXV use new DxcDllExtValidationLoader to load external validator before validation.#7749
Conversation
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
DxcDllExtValidationLoader to load external validator before validation.DxcDllExtValidationLoader to load external validator before validation.
6c69325 to
78cace7
Compare
| UINT32 newArgCount = ArgCount + 1; | ||
| std::vector<LPCWSTR> newArgs; | ||
| newArgs.reserve(newArgCount); | ||
|
|
||
| // Copy existing args | ||
| for (unsigned int i = 0; i < ArgCount; ++i) { | ||
| newArgs.push_back(pArguments[i]); | ||
| } |
There was a problem hiding this comment.
nit: I think rewriting this as below is equally performant and certainly cleaner
| UINT32 newArgCount = ArgCount + 1; | |
| std::vector<LPCWSTR> newArgs; | |
| newArgs.reserve(newArgCount); | |
| // Copy existing args | |
| for (unsigned int i = 0; i < ArgCount; ++i) { | |
| newArgs.push_back(pArguments[i]); | |
| } | |
| std::vector<LPCWSTR> newArgs = {pArguments, pArguments + ArgCount}; |
| if (pResult == nullptr) | ||
| return E_POINTER; |
There was a problem hiding this comment.
nit:
| if (pResult == nullptr) | |
| return E_POINTER; | |
| if (!pResult) | |
| return E_POINTER; |
| hr = evh->QueryInterface(riid, reinterpret_cast<void **>(pResult)); | ||
| return hr; | ||
|
|
||
| } else if (clsid == CLSID_DxcValidator) { |
There was a problem hiding this comment.
nit: you can delete the else here. Might even be worth moving the entire if to the top of this block since its quite short comparted to the other one
| IUnknown **pResult) { | ||
| if (DxilExtValSupport.IsEnabled() && clsid == CLSID_DxcValidator) | ||
| return DxilExtValSupport.CreateInstance2(pMalloc, clsid, riid, pResult); | ||
| if (pResult == nullptr) |
There was a problem hiding this comment.
nit:
| if (pResult == nullptr) | |
| if (!pResult) |
| if (DXC_FAILED(ValHR)) { | ||
| if (SUCCEEDED(pCompileResult->QueryInterface(&pResult))) |
There was a problem hiding this comment.
nit: it looks like these can be a single if statement
| // validation scenario, so there is no point in also running | ||
| // the internal validator. This wrapper wraps both the | ||
| // IDxcCompiler and IDxcCompiler3 interfaces. | ||
| class ExternalValidationHelper : public IDxcCompiler, public IDxcCompiler3 { |
There was a problem hiding this comment.
For one, you'll need to wrap all the interfaces that the compiler object implements so you don't lose the object wrapping for any of them.
But also, IDxcCompiler and IDxcCompiler3 should be incompatible to derive from and implement in the same class due to the nature of COM not supporting separate overloaded methods of the same name. This requires a separate object to implement the other interfaces.
There was a problem hiding this comment.
So to be clear, are you suggesting one parent class, ExternalValidationHelper, that implements 4 interfaces and has 4 wrapper member objects, that each implement one interface: IDxcCompiler, IDxcCompiler2, IDxcCompiler3, and IDxcValidator? Your comment above, "This code should all be inside the compiler object wrapper.", makes me think that this class should also implement the IDxcValidator interface, is that correct?
There was a problem hiding this comment.
But also,
IDxcCompilerandIDxcCompiler3should be incompatible to derive from and implement in the same class due to the nature of COM not supporting separate overloaded methods of the same name.
It's lucky that IDxcCompiler and IDxcCompiler3 don't actually have any identical methods in them so that this isn't a problem.
IDxcCompiler::Compile takes 10 args, while IDxcCompiler3::Compile takes 7
Preprocess only exists on IDxcCompiler
IDxcCompiler::Disassemble takes 2 args while IDxcCompiler3::Disassemble take 3.
If you only try and implement one of these when you derive from both interfaces you'll get a warning about hiding the base class version.
There was a problem hiding this comment.
Interesting. I thought it was not allowed, hence the hoops we jumped through multiple times for multiple implementations of the compiler object supporting both interfaces to avoid deriving from both on the same implementation class.
In any case the first point stands, that there are more interfaces to support than these two.
You'll want to derive from each of these:
IDxcCompiler2IDxcCompiler3IDxcLangExtensions3IDxcContainerEventIDxcVersionInfo3IDxcVersionInfo2(orIDxcVersionInfoifSUPPORT_QUERY_GIT_COMMIT_INFOnot defined)
And no need to derive from IDxcCompiler when deriving from IDxcCompiler2, since IDxcCompiler2 already derives from IDxcCompiler.
There was a problem hiding this comment.
The implementation in this PR will cause failures if any of the tests attempt to use these interfaces. I don't think we need to wrap additional ones for the sake of it.
There was a problem hiding this comment.
Like Damyan said, I think I'll worry about deriving from each of those in a future PR.
There was a problem hiding this comment.
There are quite a few issues, and as I tried to write something that could express the exact logic necessary for the arguments, I started rewriting the ExternalValidationHelper implementation to also fix many of the other issues. I put a branch up, so here's the link to the commit:
See the commit for the general list of issues this rewrite fixes.
c6ac790 to
eb69e26
Compare
alsepkow
left a comment
There was a problem hiding this comment.
Submitting a block of comments while I continue to review.
| if (Result) | ||
| return Result->QueryInterface(Riid, ValResult); | ||
| else | ||
| return E_FAIL; |
There was a problem hiding this comment.
I suggest returning E_POINTER as I think that's what this HRESULT is intended to reflect?
That is, we expect that the caller gives us a valid IDxcOperationResult*.
|
|
||
| // No validation needed; just set the result and return. | ||
| if (!NeedsValidation) | ||
| return UseResult(CompileResult); |
There was a problem hiding this comment.
The UseResult name seems misleading to me? It really looks like it's just a glorified QueryInterface call?
I think it would be cleaner to
- Remove the UseResult lambda
- Add a check to return E_INVALID_ARG if CompileResult is an invalid pointer at the start of doValidation.
- Update line 59 to 'return CompileResult->QueryInterface(Riid, ValResuilt);
- Same update to line 67.
- Same update for line 83.
- Same update on line 87.
| if (ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor, | ||
| &ValidatorVersionMinor)) { |
There was a problem hiding this comment.
| if (ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor, | |
| &ValidatorVersionMinor)) { | |
| if (FAILED(ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor, | |
| &ValidatorVersionMinor))) { |
| #endif | ||
| !Opts.DumpDependencies && !Opts.VerifyDiagnostics && | ||
| Opts.Preprocess.empty(); | ||
| bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel; |
There was a problem hiding this comment.
| bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel; | |
| const bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel; |
| return DoBasicQueryInterface<IDxcCompiler, IDxcCompiler2, IDxcCompiler3>( | ||
| NewWrapper.p, Iid, ResultObject); | ||
| } catch (...) { | ||
| return E_FAIL; |
There was a problem hiding this comment.
I think the exception expected here would be due to an allocation failure.
So would ERROR_OUTOFMEMORY be a more appropriate HRESULT?
| if (ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor, | ||
| &ValidatorVersionMinor)) { | ||
| DXASSERT(false, "Failed to get validator version"); | ||
| return E_FAIL; |
There was a problem hiding this comment.
nit: I like to try and avoid E_FAIL because its pretty generic.
There is an 'ERROR_VERSION_PARSE_ERROR' you could use here.
There was a problem hiding this comment.
Gonna stick to E_FAIL since there is no definition for that Error code in non-windows.
Same applies to ERROR_OUTOFMEMORY
| // The ExtValidationArgHelper class helps manage arguments for external | ||
| // validation, as well as perform the validation step if needed. | ||
| class ExtValidationArgHelper { | ||
| std::vector<std::wstring> ArgStorage; |
There was a problem hiding this comment.
I think we generally do all publics first and then all privates second?
Should we move the member variables to the private section?
| // IDxcValidator interface to validate a successful compilation result, when | ||
| // validation would normally be performed. | ||
| class ExternalValidationCompiler : public IDxcCompiler2, public IDxcCompiler3 { | ||
| private: |
There was a problem hiding this comment.
nit: public first for consistency with the rest of this file
There was a problem hiding this comment.
Actually the reason this broke that consistency is because there's a special rule in linux / macos builds on initialization order, and the Compiler object can't be initialized after the m_pAlloc field, which was originally taken care of in the previous private block. So I will leave this as is.
There was a problem hiding this comment.
there's a special rule in linux / macos builds on initialization order
There's not a special rule, it's just that MSVC doesn't warn if you mismatch the order that things appear in the constructor's initialization list with the order that they are actually initialized (ie the order they appear in the class itself).
If you wanted to put the private block at the end then you'd just need to update the constructor to list m_pMalloc first (since it'd be listed first in the class as the field is added by the DXC_MICROCOM_TM_REF_FIELDS macro).
Or you could have your private block at the end look like this and not need to modify the constructor:
private:
CComPtr<IDxcValidator> Validator;
IID CompilerIID;
CComPtr<IUnknown> Compiler;
DXC_MICROCOM_TM_REF_FIELDS()| CComPtr<IDxcBlob> CompiledBlob; | ||
| IFR(CompileResult->GetResult(&CompiledBlob)); | ||
|
|
||
| // If no compiled blob; just return the compile result. |
There was a problem hiding this comment.
nit: comment doesn't seem accurate. We're returning the HRESULT of QI. That doesn't seem like a compile result?
There was a problem hiding this comment.
This language is a bit overloaded, I meant return through an out-parameter, by placing the result into ValResult.
I can rewrite this.
There was a problem hiding this comment.
The main thing that threw me off is that you wouldn't generally expect a QI call to do anything other than get you a pointer to the requested interface. The fact that the private implementation of QI is doing more than that is a little weird.
| // Return the validation result if it failed. | ||
| HRESULT HR; | ||
| IFR(TempValidationResult->GetStatus(&HR)); | ||
| if (FAILED(HR)) { |
There was a problem hiding this comment.
I see uses of DXC_FAILED and FAILED. Hadn't seen DXC_FAILED before but it looks like they do the same thing.
We should just use one.
| // function is deferred to whatever object was chosen to be pCompiler, | ||
| // which must implement the IDxcCompiler interface. External validation | ||
| // will only take place if the DXC_DXIL_DLL_PATH env var is set | ||
| // correctly. |
There was a problem hiding this comment.
Why is this comment on this one compile call? Why not put a comment where you initialize the DxcDllExtValidationLoader in dxc::main instead?
There was a problem hiding this comment.
Because the Compile command also is responsible for validation. I think this call site is appropriate because at this point, different outcomes may happen. IMO it's less clear on DxcDllExtValidationLoader initialization, since there's no compilation or validation happening then and there.
There was a problem hiding this comment.
Again, another nit I'll let go.
I guess my point was:
- This isn't the only call that could lead to external validation in DxcContext
- This isn't the code scope where that knowledge is needed or relevant - in fact, as I pointed out elsewhere,
DxcContextdoesn't need to be hard-coded to useDxcDllExtValidationLoaderin the first place.
I suggested main because that's the top-level place where options are parsed and this is a bit like an undocumented option for the tool which dxcSupport.initialize() is going to read from the environment at that point.
| private: | ||
| DxcOpts &m_Opts; | ||
| SpecificDllLoader &m_dxcSupport; | ||
| DxcDllExtValidationLoader &m_dxcSupport; |
There was a problem hiding this comment.
Shouldn't we use DllLoader instead? Isn't that the point of DllLoader as a base class?
| DxcDllExtValidationLoader &m_dxcSupport; | |
| DllLoader &m_dxcSupport; |
There was a problem hiding this comment.
The point of DllLoader is to provide abstraction for an object that is capable of creating instances of objects from a dll.
However, in the specific cases of dxc and dxv, we know ahead of time that in that context, we don't need any abstraction. dxc and dxv should both be using the more specific object type, because we know that they may both load from an external dll if the environment variable exists.
The DllLoader class was created for cases where there's delegation, and where it isn't clear whether a DxcDllExtValidationLoader instance was passed or whether something else might be passed. But we can constrain dxc and dxv to be DxcDllExtValidationLoader's only.
There was a problem hiding this comment.
I get that, and it's just a nit, so I'm fine with this.
Just to be clear why I thought we should use DllLoader, and consider it a nit:
This DxcContext is in a library and can be linked into and used from different top-level tool driver executables. If one of those doesn't need to use DxcDllExtValidationLoader and implements its own main to initialize the context, it doesn't seem right (architecturally) to force it to use DxcDllExtValidationLoader since DxcContext only needs to know about the DllLoader interface.
I'm fine with it as-is particularly because we don't plan to expand the set of tools using DxcContext. We are more likely to deprecate/remove dxl instead, which, IIRC, is the only other tools that uses this.
tex3d
left a comment
There was a problem hiding this comment.
I noticed we regressed the -external and -external-fn dxc options, and need a change to DxcDllExtValidationLoader to support this.
f69c61f to
5a3821f
Compare
tex3d
left a comment
There was a problem hiding this comment.
Approved, but would like to see the fragile __FILE__ reference from the test replaced with a reference to a file in the test data location which is required to be set for the test environment.
| template <typename T> CComPtr<T> cast() const { | ||
| CComPtr<T> Result; | ||
| if (Compiler) | ||
| Compiler->QueryInterface(__uuidof(T), reinterpret_cast<void **>(&Result)); |
There was a problem hiding this comment.
Ok, but why doesn't Result = Compiler; work? CComPtr will correctly call the QueryInterface for you upon assignment.
Even if you want to manually write the QueryInterface, you can use Compiler->QueryInterface(IID_PPV_ARGS(&Result));, or Compiler.QueryInterface(&Result); (CComPtr's templated QueryInterface).
| VERIFY_IS_TRUE(ExtSupportBogus.dxilDllFailedToLoad()); | ||
|
|
||
| // 3. Test with a valid path to this file in the environment variable | ||
| std::filesystem::path p = std::filesystem::absolute(__FILE__); |
There was a problem hiding this comment.
This presumes the test has access to this source file via the same path. That seems quite fragile, since that's not a requirement of the tests.
We do have a reliable way to access a file that must exist in the test environment: hlsl_test::GetPathToHlslDataFile(fileName) where fileName is the path to a test file relative to the tools/clang/test/HLSL path, for instance, you could use another file referenced in ValidationTest.cpp: L"..\\DXILValidation\\hs_signatures.hlsl" (through the CompileFile() function).
This PR just addresses some final concerns in #7749 It also adds a missing REQUIRES line that should've been added to a test that merged into the repo while the previous PR was in development. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This PR focuses on allowing dxc to use the existing infrastructure to use a
DxcDllExtValidationLoaderobject, and load an external dxil.dll with it, via the environment variable. The validate*.test was added to verify that the dll is indeed being loaded and used for external validation, which causes a failure, since the dll's validator version is older than the required minimum validator version that the metadata indicates.Fixes #7527